Analysis and recoding of multimodal data

Emotions are part of our lives. Emotions can enhance the meaning of our communication. However, communication with computers is still done by keyboard and mouse. In this humancomputer interaction there is no room for emotions, whereas if we would communicate with machines the way we do in face-to-face communication much information can be extracted from the context and emotion of the speaker. We have proposed a protocol for the construction of a multimodal database and a prototype that can be trained on this database for multimodal emotion recognition. The multimodal database consists of audio and videos clips for lip reading, speech analysis, vocal affect recognition, facial expression recognition and multimodal emotion recognition. We recorded these clips in a controlled environment. The purpose of this database is to make it a benchmark for the current and future emotion recognition studies in order to compare the results from different research groups. Validation of the recorded data is done online. Over 60 users scored the apex images (1.272 ratings), audio clips (201 ratings) and video clips (503 ratings) on the valence and arousal scale. Textual validation is done based on Whissell’s Dictionary of Affect in Language. A comparison is made between the scores of all four validation methods and the results showed some clusters for distinct emotions, but also some scatter for certain emotions which depend mainly on the context. Context is not always available